Combined Surface User Interface

ABSTRACT

Techniques for utilizing two or more mobile devices equipped with projectors to generate a combined seamless user interfaces by stitching projection areas generated by the projectors.

BACKGROUND

Hand-held mobile devices such as mobile phones have become small andpowerful, and they continue to develop at a rapid pace. Pocket-sized,hand-held mobile devices now have the computing power to do many thingsthat previously required large personal computers. However, small screensizes and input methods of hand-held mobile devices are stillchallenging to users, and detract from the user experience. Users desirelarger display screens to display more information and to interact moreeasily with both the user's device itself and with other devices thatmay be nearby.

SUMMARY

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used as an aid in determining the scope of the claimed subjectmatter. The term “techniques,” for instance, may refer to device(s),system(s), method(s) and/or computer-readable instructions as permittedby the context above and throughout the document.

The Detailed Description describes a mobile device that uses a projectorto illuminate a projection area external to the mobile device, and usesthe projection area as a user interface. The user interface can functionas a touch screen or multi-touch screen and a user's interaction withthe projection area can be captured by a camera of the mobile device.The mobile device can interconnect and collaborate with one or moreneighboring mobile devices to stitch their projection areas and tothereby create a combined seamless user interface that utilizes thecombined projection areas of the mobile device and its neighboringdevices. Users of the multiple mobile devices can interact with eachother on the combined seamless user interface.

BRIEF DESCRIPTION OF THE DRAWINGS

The detailed description is described with reference to the accompanyingfigures. In the figures, the left-most digit(s) of a reference numberidentifies the figure in which the reference number first appears. Thesame numbers are used throughout the drawings to reference like featuresand components.

FIG. 1 is a block diagram showing an example of how multiple mobiledevices can be used together to form a combined seamless user interface.

FIG. 2 is a block diagram showing an exemplary creation of a combinedseamless user interface by combining multiple projection areas ofmultiple mobile devices.

FIG. 3 is a block diagram showing an exemplary resource sharing scenarioamong multiple mobile devices based on the combined seamless userinterface.

FIG. 4 is a block diagram showing an exemplary cooperating work scenarioamong multiple mobile devices based on the combined seamless userinterface.

FIG. 5 is a block diagram showing an exemplary automatic identificationof neighboring devices for purposes of coupling.

FIG. 6 is a flowchart showing an exemplary procedure of stitchingoverlapping projection areas of multiple mobile devices.

DETAILED DESCRIPTION

This disclosure describes techniques for using a projector of a mobiledevice to project or illuminate a projection area external to the mobiledevice and to use the projection area as a display and user interface.The mobile device can use one or more sensing mechanisms, such as aninfrared illuminator and camera, to detect a user's interaction with theprojection area. For example, the infrared camera might be used todetect movement of a stylus or finger relative to the projection area,such as when a user touches or nearly touches the surface of theprojection area. This allows the user interface to act as a “touch” ortouch-type screen, where the user touches or nearly touches areas of theprojection area to interact with the mobile device. Both single-touchand multi-touch inputs can be detected.

In addition, two or more mobile devices can cooperate to create acombined or integrated user interface by stitching projection areas ofthe two or more mobile into an integrated and seamless display. Thecombined user interface has a larger combined display area compared tothe LCDs or other native displays of the individual mobile devices.Different users can use the combined seamless user interface to performinteractive operations such as exchanging data and workingcollaboratively on a common project, document, or other resource.

This brief introduction is provided for the reader's convenience and isnot intended to limit the scope of the claims

General Environment

FIG. 1 shows an example of how multiple mobile devices can be usedtogether to form a combined user interface. This example includes twocomputing devices, designated by reference numerals 102(a) and 102(b)and referred to as first mobile device 102(a) and second or neighboringdevice 102(b). Although the example of FIG. 1 shows only two devices,the techniques described herein can also be used with more than twodevices, to create a combined user interface using projection componentsof all such devices. One or more of the devices may also be non-mobile.

For purposes of this discussion, first mobile device 102(a) will bedescribed in some detail. Second device 102(b) and any other neighboringdevices are understood to have similar components and functionality,although they may differ significantly in some respects.

Generally, first mobile device 102(a) can be a mobile phone, a PDA, amobile internet device, a netbook, a personal media player, a laptop, ahand-held mobile device, or any other portable, mobile, computing orcommunications device.

Mobile device 102(a) can be equipped with a physical screen 104(a), aprojector 106(a), an illuminator 108(a), and one or more image or touchsensors such as a camera 110(a).

The physical screen 104(a) displays graphics to a user and can be usedas part of a default or primary graphical user interface. Screen 104(a)can be touch-sensitive to accept input from a user. Alternatively oradditionally, keys or buttons (not shown) can be utilized for user inputand interaction. The size of the physical screen 104(a) will often bequite small, limited by the small size of the mobile device 102(a).

Projector 106(a) can take many forms, including that of a so-called“pico” projector, which is small in size and has modest powerrequirements. The projector 106(a) displays a user interface on asurface 112 external to mobile device 102(a). The projected userinterface occupies or defines a projection area 114(a) on surface 112.The projected image in this embodiment can display a secondary graphicaluser interface occupying at least a portion of projection area 114(a).The user can physically interact with this secondary graphical userinterface to control or interact with the mobile device 102(a). In theexample of FIG. 1, the secondary graphical user interface fully occupiesprojection area 114(a). In many scenarios, the secondary user interfaceprovided by projector 106(a) will be much larger than the primary userinterface formed by physical screen 104(a).

Mobile device 102(a) can coordinate its physical screen 104(a) and itsexternal projection area 114(a) in different ways. In one example,physical screen 104(a) and projection area 114(a) can show the samecontent. In another example, physical screen 104(a) only shows simplecontent, such as a reminder or a clock. When the user wants a largedisplay to perform more complex or detailed operations, such as readinga document, surfing the internet, or composing an email, the user candisplay a relevant application on projection area 114(a) and interactwith the application by pointing at or touching surface 112 withinprojection area 114(a).

Illuminator 108(a) and camera 110(a) are used in combination to senseuser interaction with the projected user interface, together formingwhat will be referred to herein as an input sensor. For example,illuminator 108(a) can be an infrared emitter that illuminatesprojection area 114(a) with non-visible infrared light. More generally,illuminator 108(a) illuminates an input area 116(a) that is at least aslarge as projection area 114(a) and that encompasses projection area114(a).

Camera 110(a) can be an infrared camera, sensitive to non-visibleinfrared light incident on input area 116(a). Camera 110(a) monitors theinfrared illumination of the projection area to detect touch ortouch-like interaction by a user with the displayed user interface.Furthermore, as will be described in more detail below, camera 110(a)detects portions of projection area 114(a) that overlap with projectionareas of one or more neighboring computing devices.

There can be many different embodiments of mobile device 102(a). In oneembodiment, projector 106(a), illuminator 108(a), and camera 110(a) arebuilt into the mobile device 102(a), as shown in the FIG. 1. One or moreof projector 106(a), illuminator 108(a), and camera 110(a) can also bemodularly integrated with each other. For example, illuminator 108(a)and the camera 110(a) can be integrated as a single unit or modulewithin mobile device 102(a) or connected to mobile device 102(a).

Input area 116(a) and projection area 114(a) may or may not be exactlythe same as each other. In the example of FIG. 1, input area 116(a) islarger than and includes projection area 114(a) in order to detect userinteraction across the entire projection area 114(a). In the embodimentdescribed here, projector 106(a), illuminatorl 08(a), and camera 110(a)are preconfigured and mounted relative to each other so that when mobiledevice 102(a) is placed upright on surface 112, input area 116(a) andprojection area 114(a) are properly focused and sized on surface 112,and properly aligned with each other as shown.

In the example of FIG. 1, the secondary user interface displayed by theprojector 106(a) acts as a touch-sensitive display; the input sensorconsisting of illuminator 108(a) and camera 110(a) is able to detectwhen and where a user touches surface 112 within input area 116(a). Inthe illustrated example, the camera 110(a) senses the infraredillumination from illuminator 108(a). User interactions relative tosurface 112 cause shadows in the IR illumination, which mobile device102(a) interprets to determine placement of fingers or styli. In otherembodiments, camera 110(a) may be sensitive to visible or projectedlight to optically capture the user's interactions within the input area116(a).

Block 118 shows internal or logical components of first mobile device102(a). Second mobile device 102(b) has similar components andfunctionality. The components of mobile devices 102(a) and 102(b)include one or more processors 120, a communication system 122, andmemory 124.

Generally, memory 124 contains computer-readable instructions that areaccessible and executable by processor 112. Memory 124 may comprise avariety of computer readable storage media. Such media can be anyavailable media including both volatile and non-volatile storage media,removable and non-removable media, local media, remote media, opticalmemory, magnetic memory, electronic memory, etc.

Any number of program modules can be stored in the memory, including byway of example, an operating system, one or more applications, otherprogram modules, and program data. Each of such program modules andprogram data (or some combination thereof) may implement all or part ofthe resident components that support the data center system as describedherein.

Communication system 122 is configured to allow the first mobilecomputing device 102(a) to communicate with one or more neighboringcomputing devices. The communication system 122 can use wired orwireless techniques for communication. The neighboring computing devicescan be other mobile devices or any other computing devices, such asdigital cameras or cell phones. In order to produce a combined userinterface, neighboring devices are logically coupled with each other orotherwise connected with each other for communication, collaboration,and display coordination. Devices can be coupled either automatically,in response to physical proximity; or manually, in response to explicituser commands.

Different techniques can be used by the mobile device 102(a) toautomatically couple physically adjacent devices. For example, mobiledevice 102(a) can use wireless or bluetooth searching, ultrasonictechniques or other techniques to sense physical nearness of anotherdevice. Alternatively, mobile device 102(a) can use its camera 110(a) todetect an existence of a projection area of a neighboring device thatoverlaps projection area 114(a).

In one example, the user can manually build a connection or couplingbetween two devices such as mobile device 102(a) and neighboring device102(b). The user can input information into mobile device 102(a) thatidentifies a neighboring device to be connected. To obtain thisinformation from the user, the projector 106 (a) might project a userinput field (not shown in the FIG. 1) on the projection area 114(a)along with a projected keyboard and a prompt instructing the user toenter information about neighboring device 102(b). Such information, tobe provided by the user, might include a device name, username/password, and/or a relative position of the neighboring device102(b) to the mobile device 102(a). The user enters this informationusing the touch-sensitive or touch-like features implemented by mobiledevice 102(a) in conjunction with its projector 106(a 0, and mobiledevice 102(a) attempts to find neighboring device 102(b) according tothe information entered by the user.

In another example, the connection among multiple mobile devices can bebuilt automatically, without user action. For instance, mobile device102(a) can detect or find neighboring device 102(b) by wirelesssearching. The wireless searching can use wi-fi, infrared, bluetooth ,or other wireless techniques.

As yet another example, camera 110(a) can be monitored to detect aprojection area of another device, such as a projection area 114(b) ofneighboring device 102(b). Having sensed the existence of a neighboringdevice 102(b) in this manner, mobile device 102(a) can then attempt toconnect or couple with it.

A mobile device may request identity information of neighboring deviceor vice versa for security purposes before the two mobile devices areinterconnected or coupled for collaboration. A potential coupling can beallowed or denied based on the identity information. In one embodiment,mobile device 102(a) is configured with a “white list” of other deviceswith which coupling is allowed. This list may be configured by the userof mobile device 102(a). In another embodiment, one or more such whitelists can be maintained by a third party to which mobile device 102(a)has access. Alternatively, the user of mobile device 102(a) might beasked for confirmation prior to implementing any potential coupling.

Mobile device 102(a) also has a collaboration logic 126, which in thisexample comprises computer-executable programs, routines, orinstructions that are stored within memory 124 and are executed byprocessor 120. The collaboration logic 126 communicates with physicallyneighboring mobile devices using communication system 122, andcoordinates with those devices to graphically stitch the projection area114(a) of mobile computing device 102(a) with projection areas of one ormore neighboring computing devices. For example, as shown in FIG. 1,collaboration logic 126 stitches projection area 114(a) with projectionarea 114(b) of neighboring mobile device 102(b). Furthermore,collaboration logic 126 creates a combined seamless user interface 128utilizing a portion of projection area 114(a) of mobile computing device102(a) and a portion of projection area 114(b) of neighboring device102(b). The combined user interface 128 can be any shape or size withina boundary of both projection area 114(a) and projection area 114(b).The projection areas 114(a) and 114(b) are potentially clipped at theirintersection so that the resulting clipped projection areas do notoverlap each other, but are immediately adjacent each other. This avoidsan abnormally bright area in the combined user interface that wouldotherwise result from the overlapped illumination of the two projectors.

As shown in FIG. 1, combined user interface 128 can be larger than theuser interface that any single device might be able to display. In someusage scenarios, the collaboration logic 126 might coordinate with otherdevices to create a single user interface that is primarily forinteraction with the single mobile device 102(a), with other devicesacting in a slave mode to expand the user interface of mobile device102(b). In other usage scenarios, the combined user interface may allowconcurrent interaction with all of the different devices, andinteractions with the combined user interface may cause actions orresults in one or more of the multiple devices.

The combined seamless user interface 128 allows one or more graphicalrepresentations 130 to span and move seamlessly between the projectionsareas of the mobile computing device and the neighboring computingdevices. As an example, FIG. 1 shows a graphical representation 130 (inthis case an icon) that spans projection area 114(a) and projection area114(b). In some embodiments, graphical representation 130 can beseamlessly dragged between projection area 114(a) and projection area114(b).

Graphical representation 130 can correspond to device resources such asfiles, shortcuts, programs, documents, etc., similar to “icons” used inmany graphical operating systems to represent various resources.Graphical representation 130 might alternatively comprise a displayedresource, menu, window, pane, document, picture, or similar visualrepresentation that concurrently spans the projection areas of mobiledevice 102(a) and neighboring device 102(b).

After mobile device 102(a) is interconnected or logically coupled withneighboring device 102(b), collaboration logic 126 of mobile device102(a) can collaborate with mobile device 102(b) to create the combinedseamless user interface 128. In the example of FIG. 1 where theprojection areas 114(a) and 114(b) are overlapping, collaboration logic126 of mobile device 102(a) communicates with mobile device 102(b) tonegotiate how each device will clip portions of its projection area 114to avoid overlap. In addition, collaboration logic 126 communicatescoordinates allowing the remaining projection areas of the two devicesto be graphically stitched to each other to create the appearance of asingle projected image or user interface utilizing at least portions ofthe projection areas 114(a) and 114(b).

FIG. 2 shows an example of how the projection areas of three devicesmight be clipped and stitched to form a combined interface. FIG. 2 showsthree overlapping projection areas: projection area 114(a) produced bymobile device 102(a) of FIG. 1, projection area 114(b) produced byneighboring device 102(b), and another projection area 114(c) that mightbe produced by another neighboring device having capabilities similar tothose of mobile device 102(a). Although FIG. 2 only shows threeoverlapping projection areas, the techniques described herein can beused for creation of a combined seamless user interface by combining anynumber of overlapping projection areas.

In addition to the projection areas 114(a), 114(b), and 114(c), FIG. 2shows input area 116(a). Although other input areas are not shown inFIG. 2, it should be understood that each projection device might defineits own input area, corresponding to its projection area.

The projection area 114(a) is represented by a polygon defined by (P4,B, C, D). The projection area 114(b) is represented by a polygon definedby (P1, P2, P3, A). The projection area 114(c) is represented by apolygon defined by (E, F, P5, P6). All of the projection areas and inputareas locate at a surface 112.

As shown in FIG. 2, there is an overlapping portion between projectionarea 114(a) and projection area 114(b), forming a polygon defined by(C1, B, C2, A). Cross points between the two projection areas are C1 andC2. There is also an overlapping portion between projection area 114(a)and projection area 114(c), and cross points between the two projectionareas are C3 and C4.

Because these overlapping portions and cross points are within inputarea 116(a), camera 110(a) of mobile device 102(a) can detect theoverlapping portions and cross points. Specifically, mobile device102(a) is configured to monitor its input area 116(a) with camera 110(a)and to detect any portions of projection area 114(a) that aresignificantly brighter than other parts of the projection area 114(a),after correcting for the localized brightness of the image beingprojected by mobile device 102(a) itself in projection area 114(a).

Collaboration logic 126 of mobile device 102(a) can communicate with theneighboring devices, including neighboring device 102(b), to define anew projection area of each device in the combined seamless userinterface. Any one of the devices can detect an overlapping portion ofits projection area with a projection area of another mobile device andcalculate the cross points. In addition, any one of the devices cantransmit this information to a neighboring device, or receive thisinformation from a neighboring device. Once the cross points of theprojection areas are determined, each mobile device can graphicallystitch its projection area with the projection areas of the neighboringmobile devices to create a combined user interface.

The mobile devices communicate with each other to coordinate clipping atleast part of the detected overlapping portions from the projectionareas and stitching of remaining projection areas. In this example,mobile device 102(b) clips an overlapping portion defined by (C1, A, C2)from its projection area to leave a remaining projection area of themobile device 102(b) which is a defined by (P1, P2, P3, C2, C1). By thesame techniques, an overlapping portion defined by polygon (E, F, C3,C4) is clipped from projection area 114(c) to leave a remainingprojection defined by (C3, C4, P5, P6). Mobile device 102(a) clips anoverlapping portion defined by (C1, B, C2) and another overlappingportion defined by (C3, C4, C, D) from its projection area 114(a) toleave a remaining projection area (P4, C1, C2, C4, C3). As shown in theFIG. 2, some portions of the projection areas, such as an area definedby (F, G, C3) are also clipped from the projection area 206(c) even ifthey're not overlapping with another projection area, to preserve auniform remaining projection area. Otherwise, content displayed at theprojection area 206(c) is intersected by the remaining projection areaof the mobile device 206(b) into multiple portions, a polygon defined by(C3, C4, P5, P6) and a polygon defined by (F, G, C3).

The mobile devices communicates with each other regarding the clippedportions and the remaining projection areas, as well as any calculatedcross points. Each mobile device (e.g., mobile device 202(b)) onlyprojects graphic representations in the remaining projection area tothereby avoid overlapping.

A single display combining the three projection areas 114(a), 114(b),and 114(c) is thus created, defined by (P1, P2, P3, C2, C4, P5, P6, C3,P4, C1).

The devices then communicate in real time to create a combined seamlessuser interface combining projection areas 114(a), 114(b), and 114(c).The combined user interface occupies at least a portion of the singledisplay. In the example of FIG. 2, the combined seamless user interfacefully occupies the single display.

The shape of the remaining projection areas and the combined seamlessuser interface in FIG. 2 which is the single display represented by apolygon defined by (P1, P2, P3, C2, C4, P5, P6, C3, P4, C1) areirregular, and possibly distorted because of the angle of surface 112relative to the projectors of the various devices. Algorithms within thecollaboration logic 126 of each device calculate the shape of each ofthe clipped or remaining projection areas and map the rectangularcoordinates of each projection system to the distorted and possiblynon-rectangular coordinates of the combined user interface.

The collaboration logic of a mobile device (e.g., collaboration logic126) can also be configured to do automatic recalibration. In oneexample, if the user moves mobile device 102(b) away from mobile device102(a), mobile device 102(a) can discover this by its input sensor andthe collaboration procedures starts again to generate a new displaycombining a new projection area of mobile device 102(b) with the otherprojection areas. In another example, each mobile device routinelychecks the integrity of the combined seamless user interface. When oneprojection area is lost, the interconnection procedure will start againand a new connection will be generated. To improve the performance, itis not necessary to rebuild all the connections if there are more thanthree devices and only several mobile devices near the lost projectionarea are involved.

Although the projection areas are overlapping in FIG. 1 and FIG. 2, suchoverlapping is not necessary for the creation of the combined seamlessuser interface. In one embodiment, the mobile devices have a projectionsystem that is capable of panning and zooming Once mobile devices 102(a)and 102(b) are logically coupled, collaboration logic 126 of mobiledevice 102(a) can communicate with mobile device 102(b) to adjust a sizeor location of projection area 114(a) and/or projection area 114(b) togenerate a seamless single display by combining the projection areas ofall of the mobile devices. For example, mobile device 102(a) can act asa master device that controls projector 106(b) of mobile device 102(b)after the interconnection. Mobile device 102(b) in this example acts asa slave device that follows instructions from the master device 102(a)to adjust sizes or shapes of its projection areas 114(b). Once there isa space or overlapping between two or more projection areas, the masterdevice 102(a) can detect it through its input sensor and control its ownprojector 106(a) and/or projector 106(b) of mobile device 102(b) toadjust the location of projection areas 114(a) and 114(b) to remove thespace or the overlapping. If there are three or more projection areas tobe combined, the master device can also authorize one or more othermobile devices to control certain projection areas that the masterdevice cannot directly detect by its own sensing techniques such as acamera. In the case that the combined seamless user interface cannot becreated due to a location of one mobile device being too far away fromthe other mobile devices, such mobile device can make signals or becontrolled by the master device to make signals, such as warning sound,to remind the user to relocate it to be close to the other mobiledevices to achieve the seamless single display.

After the interconnection among multiple mobile devices is establishedand the combined seamless user interface is created, users of themultiple mobile devices can use the combined seamless user interface todisplay content, such as a movie or a slide show, or enter into a groupdiscussion mode to interact with each other on the combined seamlessuser interface.

In one embodiment, a mobile device can use the combined seamlessinterface to display content such as a movie, video, document, drawing,picture, or similar accessible resource. For example, in FIG. 2, becausea projector of the mobile device (e.g., projector 106(a)) can onlyproject a content such as a picture onto the remaining projection area(P4, C1, C2, C4, C3) of its own projection area 114(a), mobile device102(a) can act as a master device to collaborate with the neighboringdevices such as mobile device 102(b). In this situation, mobile device102(a) transmits the content to the neighboring devices and controlsthem to project a portion of the content on their respective clipped orremaining projection areas. Collaboration logic 126 of mobile device102(a) collaborates with the neighboring devices to ensure that acombination of portions of the content displayed at each remainingprojection area presents a complete representation of the content.

In another embodiment, users of the multiple mobile devices can usecombined seamless user interface to interact with each other andcollaborate with each other. Several exemplary scenarios of such groupcollaboration are described in the following section.

Collaboration Examples

FIG. 3 shows one collaboration example using a combined user interface128. In this example, the collaboration logic 126 visually representscomputing device resources of the respective computing devices on theircorresponding user interface areas. Files or other resources can betransferred between neighboring devices by dragging their visualrepresentations to different areas of the combined user interface 128.The files or other resources can comprise documents, projects, pictures,videos, shortcuts, or any other resources that can be representedgraphically on a user display.

In the example of FIG. 3, a graphical representation 302 represents aresource, such as an object containing contact information for a person.The graphical representation is shown as initially being in projectionarea 114(a), or a home area corresponding to mobile device 102(a). Theuser of the mobile device 102(a) can move graphical representation 302from the projection area 114(a), considered the home area in thisexample, to the neighboring projection area 114(c) by “dragging” it witha finger or stylus. When graphical representation 302 crosses fromprojection area 114(a) to projection area 114(b), collaboration logicwithin the two devices 102(a) and 102(b) causes the resource representedby graphical representation 302 to be moved or copied from mobile device102(a) to neighboring device 102(b).

The physical screen 104(a) of mobile device 102(a) can be used inconjunction with this technique. For example, private resources can bedisplayed on physical screen 104(a). When a user desires to share aresource, the resource can be dragged or moved onto combined userinterface 128, where it becomes visible and accessible to other users.

FIG. 4 illustrates another collaboration example in which users ofdevices 102(a) and 102(b) enter into a discussion mode to do somecooperating work. For instance, the users can finish a paintingtogether.

In this example, a drawing canvas is presented on combined userinterface 128 and one or more users can interact with the user interfaceto add features to the drawing canvas. Each user can directly paint onthe combined seamless user interface. In the illustrated example, usershave added features 402 and 404. Features can span projection areas114(a) and 114(b), as illustrated by feature 404. Features or objectscan also be moved between projection areas that form the combined userinterface. In one embodiment, any changes to the canvas are saved in alldevices that are participating in the collaboration.

Similarly, a common canvas or textual input area can be used to recordevents of a meeting. In this example, one mobile device is selected ordesignated by users to keep meeting notes of the multiple users' inputson the combined seamless user interface. In addition to the final inputson the combined seamless user interface, the meeting notes can alsotrack a history of the inputs by users, which may or may not be removedfrom the combined seamless user interface during the discussion, andinformation of which user adds a particular input at a particular time,etc.

Any user can also add some-to-do list and even a next meeting's time andlocation on the combined seamless user interface to be included in themeeting notes. When the meeting is over, the meeting notes are saved onall or selected ones of the devices participating in the meeting orcollaboration. Alternatively, the meeting record can be saved by onedevice and automatically emailed to the others.

A user can exit a group collaboration simply by removing their mobiledevice so that its projection area is separate from the combinedseamless user interface. This device then drops out of the sharedcollaboration and projection sharing. Alternatively, a user can exit byexplicit command, or by a dedicated button on the mobile device.

Optical Neighbor Recognition

FIG. 5 illustrates how neighboring devices might automatically recognizeeach other for purposes of automatic coupling. In this example, aftermobile device 102(a) has determined or detected the presence of aneighboring device by optically sensing one or more overlappingprojection areas, it uses its camera to inspect the projection area ofthe neighboring device. During this inspection, it searches for anyidentifying information that might be projected by the neighboringdevice. When attempting to connect to an adjacent device, each deviceprojects its own identifying information within its projection area. Theinformation can be in a machine-readable format such as a barcode,examples of which are shown in FIG. 5 as barcode 502 and barcode 504.The identifying information can include whatever parameters might beneeded for connecting to the device. Specifically, the information caninclude a device address, an authentication code, a user's name, and/orother information.

After the multiple mobile devices are interconnected, the multiplemobile devices collaborate to generate a combined seamless userinterface.

Exemplary Procedural Details

FIG. 6 shows an exemplary procedure 600 of collaborating between firstand second mobile computing devices. Procedure 600 is described in thecontext of the physical environment of FIG. 1, in which first and secondcomputing devices 102(a) and 102(b) have user interfaces that areprojected in respective projection areas 116(a) and 116(b) on a commonprojection surface 112. The procedure is described as being performed byfirst mobile device 102(a), although in many embodiments it will beperformed concurrently by both device 102(a) and device 102(b), andpossibly by one or more additional devices having similar capabilities.

An action 602 comprises projecting a user interface or image inprojection area 116(a) on surface 112. An action 604 comprisesilluminating projection area 116(a) with infrared light, usingilluminator 108(a) that is fixed within computing device 102(a). Anaction 606 comprises optically monitoring projection area 116(a), ormonitoring camera 110(a) and detecting overlapping parts of projectionarea 116(a), and to detect user interactions within projection area116(a).

An action 608 comprises detecting or identifying a neighboringprojection device. As discussed above, this can be accomplishedautomatically or by requesting user input. In some embodiments, thiscomprises capturing identifying information (such as a barcode)projected by neighboring device 102(b). An action 610 comprisesestablishing communications with neighboring device 102(b) andcommunicating with neighboring device 102(b) based on the capturedidentifying information.

An action 612 comprises detecting an overlapping portion of theprojection area 116(a) of first computing device 102(a) that overlapswith the projection area 116(b) of second mobile device 102(b). This canbe done by sensing or evaluating brightness of the infrared illuminationor of the projected user interface within projection area 116(a) ofcomputing device 102(a). In the embodiment described herein, it isperformed by detecting overlapping areas of infrared illumination.

An action 614 comprises communicating with second mobile device 102(b)and with other neighboring devices to coordinate clipping and stitchingof their respective projection areas. This action involves communicatingand negotiating various coordinates that define overlaps, cross points,and the clipping areas of each projection area.

An action 616 comprises clipping at least part of the detectedoverlapping portion from the projection area 114(a) of the first mobilecomputing device 102(a) to leave a remaining projection area.

An action 618 comprises graphically stitching the remaining projectionarea of the first mobile device with remaining projection areas ofneighboring devices, such as device 102(b). An action 620 comprisescoordinating with the neighboring devices to create seamless userinterface 128 that includes at least portions of the projection areas116(a) and 116(b) of first and second computing devices 102(a) and102(b) and any other neighboring devices.

CONCLUSION

Although the subject matter has been described in language specific tostructural features and/or methodological acts, it is to be understoodthat the subject matter defined in the appended claims is notnecessarily limited to the specific features or acts described. Rather,the specific features and acts are disclosed as exemplary forms ofimplementing the claims.

1. A mobile computing device comprising: a projector that displays auser interface on a surface external to the mobile computing device, theuser interface occupying a projection area on the surface; anilluminator that illuminates the projection area with non-visible light;a camera that monitors the non-visible illumination of the projectionarea to detect touch-type interaction with the displayed user interfaceand to detect portions of the projection area that overlap withprojection areas of one or more neighboring computing devices; acommunication system allowing the mobile computing device to communicatewith the one or more neighboring computing devices; collaboration logicthat interacts with the one or more neighboring computing devicesthrough the communication system to (a) graphically stitch theprojection area of the mobile computing device with projection areas ofthe one or more neighboring computing devices based on the detection ofoverlapping portions of the projection area and (b) create a combinedseamless user interface utilizing the projection area of the mobilecomputing device and the projection areas of the one or more neighboringcomputing devices; and wherein the combined seamless user interfaceallows graphical representations to span the projection areas of themobile computing device and the one or more neighboring computingdevices in response to the touch-type interaction with the combinedseamless user interface.
 2. A mobile computing device as recited inclaim 1, wherein the combined seamless user interface allows thegraphical representations to move seamlessly between the projectionareas of the mobile computing device and the one or more neighboringcomputing devices in response to the touch-type interaction with thecombined seamless user interface.
 3. A mobile computing device asrecited in claim 1, wherein the collaboration logic is configured tographically stitch the projection areas in response to physicalproximity of the one or more neighboring computing devices with themobile computing device.
 4. A mobile computing device as recited inclaim 1, wherein: the combined seamless user interface has a home areacorresponding to the mobile computing device and one or more neighboringareas corresponding to the neighboring computing devices, respectively;the collaboration logic visually represents computing device resourcesof the respective computing devices on their corresponding userinterface areas; and in response to a user moving a visualrepresentation of a particular device resource from the user interfacearea corresponding to a first of the computing devices to the userinterface area corresponding to a second of the computing devices, thecollaboration logic copies the particular device resource from the firstof the computing devices to the second of the computing devices.
 5. Amobile computing device as recited in claim 1, wherein: the camera isconfigured to capture identifying information projected by the one ormore neighboring computing devices on their projection areas; and thecollaboration logic is configured to recognize the captured identifyinginformation and in response to communicate with the one or moreneighboring computing devices and to include the projection areas of theone or more neighboring computing devices in the combined userinterface.
 6. A mobile computing device as recited in claim 1, whereinthe projection areas of the neighboring computing devices expand theuser interface of the mobile computing device.
 7. A method ofcollaborating between first and second mobile computing devices havinguser interfaces that are projected in respective projection areas on acommon projection surface, comprising: optically detecting anoverlapping portion of the projection area of the first mobile computingdevice that overlaps with the projection area of the second mobilecomputing device; clipping at least part of the detected overlappingportion from the projection area of the first mobile computing device toleave a remaining projection area of the first mobile computing device;and graphically stitching the remaining projection area of the firstmobile computing device with the projection area of the second mobilecomputing device to create a seamless user interface that includes atleast portions of the projection areas of the first and second mobilecomputing devices.
 8. A method as recited in claim 7, furthercomprising: capturing identifying information projected the secondmobile computing device; communicating with the second mobile computingdevice based on the captured identifying information.
 9. A method asrecited in claim 7, further comprising: communicating with the secondmobile computing device to coordinate clipping and stitching of theprojection areas of the first and second mobile computing devices.
 10. Amethod as recited in claim 7, wherein the first and second mobilecomputing devices illuminate their projection areas with infrared lightto detect user interaction, and wherein optically detecting theoverlapping portion comprises detecting overlapping areas of infraredlight illumination.
 11. A method as recited in claim 7, whereinoptically detecting the overlapping portion comprises sensing brightnessthe projection area of the first mobile computing device.
 12. A methodas recited in claim 7, further comprising: illuminating the projectionarea of the first mobile computing device with infrared light; andmonitoring an infrared camera to detect user interaction with theprojected user interface of the first computing device.
 13. A method asrecited in claim 7, further comprising: optically monitoring theprojected user interface of the first computing device to detect userinteraction and to detect the overlapping portion.
 14. A mobilecomputing device comprising: a projector that displays a user interfaceon a surface external to the mobile computing device, the user interfaceoccupying a projection area on the surface; an input sensor that sensesuser interaction with the displayed user interface; a communicationsystem allowing the mobile computing device to communicate with one ormore neighboring computing devices; and collaboration logic thatstitches the projection area of the mobile computing device withprojection areas of the one or more neighboring computing devices tocreate a combined user interface utilizing the projection area of themobile computing device and the projection areas of the one or moreneighboring computing devices.
 15. A mobile computing device as recitedin claim 14, wherein the collaboration logic is configured to stitch theprojection areas in response to physical proximity of the one or moreneighboring computing devices with the mobile computing device.
 16. Amobile computing device as recited in claim 14, wherein: the combineduser interface has a home area corresponding to the mobile computingdevice and one or more neighboring areas corresponding to theneighboring computing devices, respectively; the collaboration logicvisually represents computing device resources of the respectivecomputing devices on their corresponding user interface areas; and inresponse to a user moving a visual representation of a particular deviceresource from the user interface area corresponding to a first of thecomputing devices to the user interface area corresponding to a secondof the computing devices, the collaboration logic copies the particulardevice resource from the first of the computing devices to the second ofthe computing devices.
 17. A mobile computing device as recited in claim14, wherein: the input sensor comprises a camera configured to captureidentifying information projected by the one or more neighboringcomputing devices on their projection areas; and the collaboration logicis configured to recognize the captured identifying information and inresponse to communicate with the one or more neighboring computingdevices and to include the projection areas of the one or moreneighboring computing devices in the combined user interface.
 18. Amobile computing device as recited in claim 14, wherein the projectionareas of the neighboring computing devices expand the user interface ofthe mobile computing device.
 19. A mobile computing device as recited inclaim 14, wherein: the input sensor comprises a camera configured todetect physical interaction by a user; and the collaboration logic isresponsive to the camera to detect overlapping projection areas and tostitch the projection area of the mobile computing device with theprojection areas of the one or more neighboring computing devices basedon the detection of the overlapping projection areas.
 20. A mobilecomputing device as recited in claim 14, further comprising: anilluminator that illuminates the projection area of the mobile computingdevice with infrared light; and the input sensor comprising a camerathat monitors the infrared illumination of the projection area of themobile computing device to detect physical interaction by a user.